Method of and System for Using Conversation State Information in a Conversational Interaction System

ABSTRACT

A method of using conversation state information in a conversational interaction system is disclosed. A method of inferring a change of a conversation session during continuous user interaction with an interactive content providing system includes receiving input from the user including linguistic elements intended by the user to identify an item, associating a linguistic element of the input with a first conversation session, and providing a response based on the input. The method also includes receiving additional input from the user and inferring whether or not the additional input from the user is related to the linguistic element associated with the conversation session. If related, the method provides a response based on the additional input and the linguistic element associated with the first conversation session. Otherwise, the method provides a response based on the second input without regard for the linguistic element associated with the first conversation session.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S.application Ser. No. 13/667,388 entitled “Method of and System for UsingConversation State Information in a Conversational Interaction System,”filed on Nov. 2, 2012, which claims priority under 35 U.S.C. §119(e) toU.S. Provisional Patent Application No. 61/673,867 entitled “AConversational Interaction System for Large Corpus InformationRetrieval,” filed on Jul. 20, 2012, and U.S. Provisional PatentApplication No. 61/712,721 entitled “Method of and System for ContentSearch Based on Conceptual Language Clustering,” filed on Oct. 11, 2012,the entire contents of each of which are incorporated by referenceherein.

BACKGROUND OF THE INVENTION

1. Field of Invention

The invention generally relates to conversational interactiontechniques, and, more specifically, to inferring user input intent basedon resolving input ambiguities and/or inferring a change inconversational session has occurred.

2. Description of Related Art

Conversational systems are poised to become a preferred mode ofnavigating large information repositories across a range of devices:Smartphones, Tablets, TVs/STBs, multi-modal devices such as wearablecomputing devices such as “Goggles” (Google's sunglasses), hybridgesture recognition/speech recognition systems like Xbox/Kinect,automobile information systems, and generic home entertainment systems.The era of touch based interfaces being center stage, as the primarymode of interaction, is perhaps slowly coming to an end, where in manydaily life use cases, user would rather speak his intent, and the systemunderstands and executes on the intent. This has also been triggered bythe significant hardware, software and algorithmic advances making textto speech significantly effective compared to a few years ago.

While progress is being made towards pure conversation interfaces,existing simple request response style conversational systems sufficeonly to addresses specific task oriented or specific informationretrieval problems in small sized information repositories—these systemsfail to perform well on large corpus information repositories.

Current systems that are essentially request response systems at theircore, attempt to offer a conversational style interface such asresponding to users question, as follows:

-   -   User: What is my checking account balance?    -   System: It is $2,459.34.    -   User: And savings?    -   System: It is $6,209.012.    -   User: How about the money market?    -   System: It is $14,599.33.

These are inherently goal oriented or task oriented request responsesystems providing a notion of continuity of conversation though eachrequest response pair is independent of the other and the only contextmaintained is the simple context that it is user's bank account. Otherexamples of current conversational systems are ones that walk userthrough a sequence of well-defined and often predetermined decision treepaths, to complete user intent (such as making a dinner reservation,booking a flight etc.)

Applicants have discovered that understanding user intent (even within adomain such as digital entertainment where user intent could span frompure information retrieval, to watching a show, or reserving a ticketfor a show/movie), combined with understanding the semantics of the userutterance expressing the intent, so as to provide a clear and succinctresponse matching user intent is a hard problem that present systems inthe conversation space fall short in addressing. Barring simplesentences with clear expression of intent, it is often hard to extractintent and the semantics of the sentence that expresses the intent, evenin a single request/response exchange style interaction. Adding to thiscomplexity, are intents that are task oriented without having welldefined steps (such as the traversal of a predetermined decision tree).Also problematic are interactions that require a series of user requestsand system responses to get to the completion of a task (e.g., likemaking a dinner reservation). Further still, rich informationrepositories can be especially challenging because user intentexpression for an entity may take many valid and natural forms, and thesame lexical tokens (words) may arise in relation to many different userintents.

When the corpus is large, lexical conflict or multiple semanticinterpretations add to the complexity of satisfying user intent withouta dialog to clarify these conflicts and ambiguities. Sometimes it maynot even be possible to understand user intent, or the semantics of thesentence that expresses the intent—similar to what happens in real lifeconversations between humans. The ability of the system to ask theminimal number of questions (from the point of view of comprehending theother person in the conversation) to understand user intent, just like ahuman would do (on average where the participants are both aware of thedomain being discussed), would define the closeness of the system tohuman conversations.

Systems that engage in a dialog or conversation, which go beyond thesimple multi-step travel/dinner reservation making (e.g., where thesteps in the dialog are well defined request/response subsequences withnot much ambiguity resolution in each step), also encounter thecomplexity of having to maintain the state of the conversation in orderto be effective. For example, such systems would need to infer implicitreferences to intents and entities (e.g., reference to people, objectsor any noun) and attributes that qualify the intent in user's sentences(e.g., “show me the latest movies of Tom Hanks and not the old ones;“show me more action and less violence). Further still, applicants havediscovered that it is beneficial to track not only references made bythe user to entities, attributes, etc. in previous entries, but also toentities, attributes, etc. of multi-modal responses of the system to theuser.

Further still, applicants have found that maintaining pronoun toobject/subject associations during user/system exchanges enhances theuser experience. For example, a speech analyzer (or natural languageprocessor) that relates the pronoun “it” to its object/subject “LedZeppelin song” in a complex user entry, such as, “The Led Zeppelin songin the original sound track of the recent Daniel Craig movie . . . Whoperformed it?” assists the user by not requiring the user to always usea particular syntax. However, this simple pronoun to object/subjectassociation is ineffective in processing the following exchange:

-   -   Q1: Who acts as Obi-wan Kenobi in the new star wars?    -   A: Ewan McGregor.    -   Q2: How about his movies with Scarlet Johansson?

Here the “his” in the second question refers to the person in theresponse, rather than from the user input. A more complicated examplefollows:

-   -   Q1: Who played the lead roles in Kramer vs. Kramer?    -   A1: Meryl Streep and Dustin Hoffman.    -   Q2: How about more of his movies?    -   A2: Here are some of Dustin Hoffman movies . . . [list of Dustin        Hoffman movies].    -   Q3: What about more of her movies?

Here the “his” in Q2 and “her” in Q3 refer back to the response A1. Anatural language processor in isolation is ineffective in understandinguser intent in these cases. In several of the embodiments describedbelow, the language processor works in conjunction with a conversationstate engine and domain specific information indicating male and femaleattributes of the entities that can help resolve these pronounreferences to prior conversation exchanges.

Another challenge facing systems that engage a user in conversation isthe determination of the user's intent change, even if it is within thesame domain. For example, user may start off with the intent of findingan answer to a question, e.g., in the entertainment domain. Whileengaging in the conversation of exploring more about that question,decide to pursue a completely different intent path. Current systemsexpect user to offer a clear cue that a new conversation is beinginitiated. If the user fails to provide that important clue, the systemresponses would be still be constrained to the narrow scope of theexploration path user has gone down, and will constrain users input tothat narrow context, typically resulting undesirable, if not absurd,responses. The consequence of getting the context wrong is even moreglaring (to the extent that the system looks comically inept) when userchooses to switch domains in the middle of a conversation. For instance,user may, while exploring content in the entertainment space, say, “I amhungry”. If the system does not realize this as a switch to a new domain(restaurant/food domain), it may respond thinking “I am hungry” is aquestion posed in the entertainment space and offer responses in thatdomain, which in this case, would be a comically incorrect response.

A human, on the other hand, naturally recognizes such a drastic domainswitch by the very nature of the statement, and responds accordingly(e.g., “Shall we order pizza?”). Even in the remote scenario where thetransition to new domain is not so evident, a human participant mayfalter, but quickly recover, upon feedback from the first speaker (“Ohno. I mean I am hungry—I would like to eat!”). These subtle, yetsignificant, elements of a conversation, that humans take for granted inconversations, are the ones that differentiate the richness ofhuman-to-human conversations from that with automated systems.

In summary, embodiments of the techniques disclosed herein attempt toclosely match user's intent and engage the user in a conversation notunlike human interactions. Certain embodiments exhibit any one or moreof the following, non-exhaustive list of characteristics: a) resolveambiguities in intent and/or description of the intent and, wheneverapplicable, leverage off of user's preferences (some implementations usecomputing elements and logic that are based on domain specific verticalinformation); b) maintain state of active intents and/orentities/attributes describing the intent across exchanges with theuser, so as to implicitly infer references made by user indirectly tointents/entities/attributes mentioned earlier in a conversation; c)tailor responses to user, whenever applicable, to match user'spreferences; d) implicitly determine conversation boundaries that starta new topic within and across domains and tailor a response accordingly;e) given a failure to understand user's intent (e.g., either because theintent cannot be found or the confidence score of its best guess isbelow a threshold), engage in a minimal dialog to understand user intent(in a manner similar to that done by humans in conversations tounderstand intent.) In some embodiments of the invention, theunderstanding of the intent may leverage off the display capacity of thedevice (e.g., like a tablet device) to graphically display intuitiverenditions that user could interact with to offer clues on user intent.

BRIEF SUMMARY OF THE INVENTION

In one aspect of the invention, a method of and system for usingconversation state information in a conversational interaction system isprovided.

In another aspect of the invention, a method of inferring a change of aconversation session during continuous user interaction with aninteractive content providing system includes providing access to a setof content items. Each of the content items has associated metadata thatdescribes the corresponding content item. The method also includesreceiving a first input from the user. The first input includeslinguistic elements intended by the user to identify at least onedesired content item. The method further includes associating at leastone linguistic element of the first input with a first conversationsession and providing a first response based on the first input andbased on the metadata associated with the content items. The method alsoincludes receiving a second input from the user and inferring whether ornot the second input from the user is related to the at least onelinguistic element associated with the first conversation session. Upona condition in which the second input is inferred to relate to the atleast one linguistic element associated with the first conversationsession, the method calls for providing a second response based on themetadata associated with the content items, the second input, and the atleast one linguistic element of the first input associated with thefirst conversation session. Upon a condition in which the second inputis inferred to not relate to the at least one linguistic elementassociated with the first conversation session, the method calls forproviding a second response based on the metadata associated with thecontent items and the second input.

In a further aspect of the invention, the inferring whether or not thesecond input from the user is related to the at least one linguisticelement associated with the first conversation session is based on adetermination of whether or not the second input contains a linguisticlinking element.

In another aspect of the invention, the inferring concludes the secondinput from the user is related upon a condition in which the secondinput contains a linguistic linking element.

In still a further aspect of the invention, the inferring concludes thesecond input from the user is not related upon a condition in which thesecond input does not contain a linguistic linking element.

In yet another aspect of the invention, the method also includes, upon acondition in which the second input does not contain a linguisticlinking element: determining a measure of relatedness between (i)linguistic elements of the second input and (ii) the at least onelinguistic element associated with the first conversation session basedon the metadata associated with the content items. Upon a condition inwhich the measure of relatedness is equal to or greater than a thresholdvalue, the inferring concludes the second input is related to the atleast one linguistic element associated with the first conversationsession. Upon a condition in which the measure of relatedness is lessthan the threshold value, the inferring concludes the second input isnot related to the at least one linguistic element associated with thefirst conversation session.

In an aspect of the invention, the metadata associated with the contentitems includes a mapping of relationships between entities associatedwith the content items. The determining the measure of relatednessincludes analyzing the mapping of relationships.

In still another aspect of the invention, the inferring whether or notthe second input from the user is related to the at least one linguisticelement associated with the first conversation session includes:identifying a linguistic element associated with the first conversationsession that identifies at least one entity, identifying a linguisticlinking element of the second input, and determining whether or not thelinguistic linking element of the second input is a suitable link to thelinguistic element associated with the first conversation session thatidentifies at least one entity. Upon a condition in which the linguisticlinking element of the second input is a suitable link, the methodconcludes the second input from the user is related to the at least onelinguistic element associated with the first conversation session. Upona condition in which the linguistic linking element of the second inputis not a suitable link, the method concludes the second input from theuser is not related to the at least one linguistic element associatedwith the first conversation session.

In a further aspect of the invention, the inferring whether or not thesecond input from the user is related to the at least one linguisticelement associated with the first conversation session includesdetermining whether or not the second input includes a linguisticelement that identifies at least one entity. Upon a condition in whichthe second input does not include a linguistic element that identifiesat least one entity, the method determines whether or not the secondinput is an appropriate response to the first response. Upon a conditionin which the second input is an appropriate response to the firstresponse, the method concludes the second input from the user is relatedto the at least one linguistic element associated with the firstconversation session. Upon a condition in which the second input is notan appropriate response to the first response, the method concludes thesecond input from the user is not related to the at least one linguisticelement associated with the first conversation session.

In yet a another aspect of the invention, upon the condition in whichthe second input is inferred to relate to the at least one linguisticelement associated with the first conversation session, the providingthe second response includes substituting the at least one linguisticelement associated with the first conversation session in place of atleast one linguistic element of the second input.

In another aspect of the invention, the method also includes associatingat least one linguistic element of the first response with the firstconversation session. Upon a condition in which the second input isinferred to relate to the at least one linguistic element associatedwith the first conversation session, the method includes further basingthe second response on the at least one linguistic element of the firstresponse associated with the first conversation session.

In still a further aspect of the invention, upon the condition in whichthe second input is inferred to relate to the at least one linguisticelement associated with the first conversation session, the methodprovides the second response including substituting the at least onelinguistic element associated with the first conversation session inplace of at least one linguistic element of the second input.

In yet another aspect of the invention, the method also includesdetermining that a portion of at least one of the first input from theuser and the second input from the user contains an ambiguousidentifier. The ambiguous identifier intended by the user to identify,at least in part, the at least one desired content item. The method alsoincludes inferring a meaning for the ambiguous identifier based onmatching portions of the at least one of the first input from the userand the second input from the user to preferences of the user describedby a user preference signature. The providing a second responseincluding selecting content items from the set of content items based oncomparing the inferred meaning of the ambiguous identifier with metadataassociated with the content items.

Any of the aspects listed above can be combined with any of the otheraspects listed above and/or with the techniques disclosed herein.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

For a more complete understanding of various embodiments of the presentinvention, reference is now made to the following descriptions taken inconnection with the accompanying drawings in which:

FIG. 1 illustrates a user interface approach incorporated here forelucidative purposes.

FIG. 2 illustrates a user interface approach incorporated here forelucidative purposes.

FIG. 3 illustrates a user interface approach incorporated here forelucidative purposes.

FIG. 4 illustrates a user interface approach incorporated here forelucidative purposes.

FIG. 5 illustrates a user interface approach incorporated here forelucidative purposes.

FIG. 6 illustrates an example of a graph that represents entities andrelationships between entities.

FIG. 7 illustrates an example of a graph that represents entities andrelationships between entities.

FIG. 8 illustrates an example of a graph that represents entities andrelationships between entities.

FIG. 9 illustrates an example of a graph that represents entities andrelationships between entities.

FIG. 10 illustrates an architecture that is an embodiment of the presentinvention.

FIG. 11 illustrates a simplified flowchart of the operation ofembodiments of the invention.

FIG. 12 illustrates a control flow of the operation of embodiments ofthe invention.

DETAILED DESCRIPTION

Preferred embodiments of the invention include methods of and systemsfor inferring user's intent and satisfying that intent in aconversational exchange. Certain implementations are able to resolveambiguities in user input, maintain state of intent, entities, and/orattributes associated with the conversational exchange, tailor responsesto match user's preferences, infer conversational boundaries that starta new topic (i.e. infer a change of a conversational session), and/orengage in a minimal dialog to understand user intent. The concepts thatfollow are used to describe embodiments of the invention.

Information Repositories

Information repositories are associated with domains, which aregroupings of similar types of information and/or certain types ofcontent items. Certain types of information repositories includeentities and relationships between the entities. Eachentity/relationship has a type, respectively, from a set of types.Furthermore, associated with each entity/relationship are a set ofattributes, which can be captured, in some embodiments, as a definedfinite set of name-value fields. The entity/relationship mapping alsoserves as a set of metadata associated with the content items becausethe entity/relationship mapping provides information that describes thevarious content items. In other words, a particular entity will haverelationships with other entities, and these “other entities” serve asmetadata to the “particular entity”. In addition, each entity in themapping can have attributes assigned to it or to the relationships thatconnect the entity to other entities in the mapping. Collectively, thismakes up the metadata associated with the entities/content items. Ingeneral, such information repositories are called structured informationrepositories. Examples of information repositories associated withdomains follow below.

A media entertainment domain includes entities, such as, movies,TV-shows, episodes, crew, roles/characters, actors/personalities,athletes, games, teams, leagues and tournaments, sports people, musicartists and performers, composers, albums, songs, news personalities,and/or content distributors. These entities have relationships that arecaptured in the information repository. For example, a movie entity isrelated via an “acted in” relationship to one or more actor/personalityentities. Similarly, a movie entity may be related to an music albumentity via an “original sound track” relationship, which in turn may berelated to a song entity via a “track in album” relationship. Meanwhile,names, descriptions, schedule information, reviews, ratings, costs, URLsto videos or audios, application or content store handles, scores, etc.may be deemed attribute fields.

A personal electronic mail (email) domain includes entities, such as,emails, email-threads, contacts, senders, recipients, company names,departments/business units in the enterprise, email folders, officelocations, and/or cities and countries corresponding to officelocations. Illustrative examples of relationships include an emailentity related to its sender entity (as well as the to, cc, bcc,receivers, and email thread entities.) Meanwhile, relationships betweena contact and his or her company, department, office location can exist.In this repository, instances of attribute fields associated withentities include contacts' names, designations, email handles, othercontact information, email sent/received timestamp, subject, body,attachments, priority levels, an office's location information, and/or adepartment's name and description.

A travel-related/hotels and sightseeing domain includes entities, suchas, cities, hotels, hotel brands, individual points of interest,categories of points of interest, consumer facing retail chains, carrental sites, and/or car rental companies. Relationships between suchentities include location, membership in chains, and/or categories.Furthermore, names, descriptions, keywords, costs, types of service,ratings, reviews, etc. all amount of attribute fields.

An electronic commerce domain includes entities, such as, product items,product categories and subcategories, brands, stores, etc. Relationshipsbetween such entities can include compatibility information betweenproduct items, a product “sold by” a store, etc. Attribute fields ininclude descriptions, keywords, reviews, ratings, costs, and/oravailability information.

An address book domain includes entities and information such as contactnames, electronic mail addresses, telephone numbers, physical addresses,and employer.

The entities, relationships, and attributes listed herein areillustrative only, and are not intended to be an exhaustive list.

Embodiments of the present invention may also use repositories that arenot structured information repositories as described above. For example,the information repository corresponding to network-based documents(e.g., the Internet/World Wide Web) can be considered a relationship webof linked documents (entities). However, in general, no directlyapplicable type structure can meaningfully describe, in a nontrivialway, all the kinds of entities and relationships and attributesassociated with elements of the Internet in the sense of the structuredinformation repositories described above. However, elements such asdomain names, internet media types, filenames, filename extension, etc.can be used as entities or attributes with such information.

For example, consider a corpus consisting of a set of unstructured textdocuments. In this case, no directly applicable type structure canenumerate a set of entities and relationships that meaningfully describethe document contents. However, application of semantic informationextraction processing techniques as a pre-processing step may yieldentities and relationships that can partially uncover structure fromsuch a corpus.

Illustrative Examples of Accessing Information Repositories UnderCertain Embodiments of the Present Invention

The following description illustrates examples of information retrievaltasks in the context of structured and unstructured informationrepositories as described above.

In some cases, a user is interested in one or more entities of sometype—generally called intent type herein—which the user wishes touncover by specifying only attribute field constraints that the entitiesmust satisfy. Note that sometimes intent may be a (type, attribute) pairwhen the user wants some attribute of an entity of a certain type. Forexample, if the user wants the rating of a movie, the intent could beviewed as (type, attribute)=(movie, rating). Such query-constraints aregenerally called attribute-only constraints herein.

Whenever the user names the entity or specifies enough information todirectly match attributes of the desired intent type entity, it is anattribute-only constraint. For example, when the user identifies a movieby name and some additional attribute (e.g., ‘Cape Fear’ made in the60s), or when he specifies a subject match for the email he wants touncover, or when he asks for hotels based on a price range, or when hespecifies that he wants a 32 GB, black colored iPod touch.

However, in some cases, a user is interested in one or more entities ofthe intent type by specifying not only attribute field constraints onthe intent type entities but also by specifying attribute fieldconstraints on or naming other entities to which the intent typeentities are connected via relationships in some well defined way. Suchquery-constraints are generally called connection oriented constraintsherein.

An example of a connection oriented constraint is when the user wants amovie (an intent type) based on specifying two or more actors of themovie or a movie based on an actor and an award the movie won. Anotherexample, in the context of email, is if the user wants emails (intenttype) received from certain senders from a particular company in thelast seven days. Similarly, a further example is if the user wants tobook a hotel room (intent type) to a train station as well as aStarbucks outlet. Yet another example is if the user wants a televisionset (intent type) made by Samsung that is also compatible with aNintendo Wii. All of these are instances of connection orientedconstraints queries.

In the above connection-oriented constraint examples, the userexplicitly describes or specifies the other entities connected to theintent entities. Such constraints are generally called explicitconnection oriented constraints and such entities as explicit entitiesherein.

Meanwhile, other queries contain connection oriented constraints thatinclude unspecified or implicit entities as part of the constraintspecification. In such a situation, the user is attempting to identify apiece of information, entity, attribute, etc. that is not known throughrelationships between the unknown item and items the user does now. Suchconstraints are generally called implicit connection orientedconstraints herein and the unspecified entities are generally calledimplicit entities of the constraint herein.

For example, the user may wish to identify a movie she is seeking vianaming two characters in the movie. However, the user does not recallthe name of one of the characters, but she does recall that a particularactor played the character. Thus, in her query, she states one characterby name and identifies the unknown character by stating that thecharacter was played by the particular actor.

In the context of email repository, an example includes a user wantingto get the last email (intent) from an unspecified gentleman from aspecified company ‘Intel’ to whom he was introduced via email (anattribute specifier) last week. In this case, the implicit entity is acontact who can be discovered by examining contacts from ‘Intel’, via anemployee/company relationship, who was a first timecommon-email-recipient with the user last week.

Further examples of implicit connection oriented constraints aredescribed in more detail below.

In the context of connection oriented constraints, it is useful to mapentities and relationships of information repositories to nodes andedges of a graph. The motivation for specifically employing the graphmodel is the observation that relevance, proximity, and relatedness innatural language conversation can be modeled simply by notions such aslink-distance and, in some cases, shortest paths and smallest weighttrees. During conversation when a user dialog involves other entitiesrelated to the actually sought entities, a subroutine addressinginformation retrieval as a simple graph search problem effectively helpsreduce dependence on deep unambiguous comprehension of sentencestructure. Such an approach offers system implementation benefits. Evenif the user intent calculation is ambiguous or inconclusive, so long asentities have been recognized in the user utterance, agraph-interpretation based treatment of the problem enables a system torespond in a much more intelligible manner than otherwise possible, asset forth in more detail below.

Attribute-Only Constraints

What follows are examples of information retrieval techniques thatenable the user to specify attribute-only constraints. While some ofthese techniques are known in the art (where specified), the conceptsare presented here to illustrate how these basic techniques can be usedwith the inventive techniques described herein to enhance the userexperience and improve the quality of the search results that arereturned in response to the user's input.

Examples of Attributes-Only Constraints During Information Retrievalfrom a Movie/TV Search Interface

FIG. 1 shows a search interface 100 for a search engine for movie andtelevision content that is known in the art (i.e., the IMDb searchinterface). FIG. 1 includes a pull-down control 105 that allows the userto expressly select an entity type or attribute. For example, Titlemeans intent entity type is Movie or TV Show, TV Episode means theintent type is Episode, Names means intent type is Personality,Companies means the intent type is Company (e.g., Production house orStudio etc.), Characters means the intent type is Role. Meanwhile,Keywords, Quotes, and Plots specify attribute fields associated withintent entities of type Movie or TV Show or Episode that are sought tobe searched. Meanwhile, the pull-down control 110 allows the user toonly specify attributes for entities of type Movie, Episode, or TV Show.

FIG. 2 shows the Advanced Title Search graphical user interface of theIMDB search interface (known in the art) 200. Here, the Title Typechoice 205 amounts to selection of intent entity type. Meanwhile,Release Date 210, User Rating 215, and Number of Votes 220 are allattributes of entities of type movies, TV Shows, episodes, etc. If thenumber of Genres 225 and Title Groups 230 shown here is deemed smallenough, then those genres and title groups can be deemed descriptiveattributes of entities. So the genre and title groups section here isalso a way of specifying attribute constraints. The Title Data 235section is specifying the constraint corresponding to the data sourceattribute.

Examples of Attributes-Only Constraints During Information Retrievalfrom an Electronic-Commerce Search Interface

FIG. 3 illustrates a graphical user interface 300 for anelectronic-commerce website's search utility that is known in the art.In previous examples, the user interface allowed users to specify setsof attribute constraints before initiating any search in the informationrepository. Meanwhile, FIG. 3 shows the user interface after the userhas first launched a text-only search query ‘car stereo’. Based onfeatures and attributes associated with the specific results returned bythe text search engine for the text search query 305, the post-searchuser interface is constructed by dynamically picking a subset ofattributes for this set of search results, which allows the user tospecify further attribute constraints for them. As a result, the user isforced to follow the specific flow of first doing a text search orcategory filtering and then specifying the constraints on furtherattributes.

This ‘hard-coded’ flow—of first search followed by post-search attributefilters—results from a fundamental limitation of this style of graphicaluser interface because it simply cannot display all of the meaningfulattributes up-front without having any idea of the product the user hasin mind. Such an approach is less efficient that the inventivetechniques disclosed herein because the user may want to declare some ofthe attribute constraints he or she has in mind at the beginning of thesearch. This problem stems, in part, from the fact that even though thenumber of distinct attributes for each individual product in thedatabase is a finite number, the collective set is typically largeenough that a graphical user interface cannot display a sufficientnumber of the attributes, thereby leading to the hard coded flow.

Note that the conversational interface embodiments disclosed herein donot suffer from physical spatial limitations. Thus, a user can easilyspecify any attribute constraint in the first user input.

Explicit Connection Oriented Constraints

What follows are examples of explicit connection oriented constraintsemployed in information retrieval systems. Graph model terminology ofnodes and edges can also be used to describe connection orientedconstraints as can the terminology of entities and relationships.

When using an attribute-only constraints interface, the user onlyspecifies the type and attribute constraints on intent entities.Meanwhile, when using an explicit connected node constraints interface,the user can additionally specify the type and attribute constraints onother nodes connected to the intent nodes via specified kinds of edgeconnections. One example of an interface known in the art that employsexplicit connected node constraints during information retrieval is aMovie/TV information search engine 400 shown in FIG. 4.

Considering that the number of possible death and birth places 405across all movie and TV personalities is a huge number, birth and deathplaces are treated as nodes rather than attributes in the movieinformation repository graph. Thus, birth and death place specificationsin the graphical user interface 400 are specifications for nodesconnected to the intended personality node. The filmography filter 410in the graphical user interface 400 allows a user to specify the name ofa movie or TV show node, etc., which is again another node connected tothe intended personality node. The other filters 500, shown in FIG. 5,of the graphical user interface are specifiers of the attributes of theintended node.

In the first part of the graphical user interface 400, a user mayspecify two movie or TV show nodes when his intent is to get thepersonalities who collaborated on both these nodes. In the second partof the graphical UI above, a user may specify two personality nodes whenhis intent is to get movie or TV show nodes corresponding to theircollaborations. In both case, the user is specifying connected nodesother than his intended nodes, thereby making this an explicit connectednode constraint. However, the interfaces known in the art do not supportcertain types of explicit connected node constraints (explicitconnection oriented constraints), as described below.

FIG. 6 illustrates a graph 600 of the nodes (entities) and edges(relationships) analyzed by the inventive techniques disclosed herein toarrive at the desired result when the user seeks a movie based on thefictional character Jack Ryan that stars Sean Connery. The user mayprovide the query, “What movie has Jack Ryan and stars Sean Connery?”The techniques herein interpret the query, in view of the structuredinformation repositories as: Get the node of type Movie (intent) that isconnected by an edge 605 to the explicit node of type Role named ‘JackRyan’ 610 and also connected via an ‘Acted In’ edge 615 to the explicitnode of type Personality named ‘Sean Connery’ 620. The techniquesdescribed herein return the movie ‘The Hunt for the Red October’ 625 asa result.

Referring again to FIG. 6, assume the user asks, “Who are all of theactors that played the character of Jack Ryan?” The disclosed techniqueswould interpret the query as:

-   -   Get nodes of type Personality (intent) connected by means of an        ‘Acted-as’ edge 630 to the explicit node of type Role named        ‘Jack Ryan’ 610. Embodiments of the inventive systems disclosed        herein would return the actors ‘Alec Baldwin’ 635, ‘Harrison        Ford’ 640, and ‘Ben Affleck’ 645.

A further example is a user asking for the name of the movie starringTom Cruise based on a John Grisham book. Thus, the query becomes: Getthe node of type Movie (intent) connected by an ‘Acted In’ edge to theexplicit node of type Personality named Tom Cruise and connected by a‘Writer’ edge to the explicit node of type Personality named ‘JohnGrisham’. Embodiments of the inventive systems disclosed herein wouldreturn the movie ‘The Firm’.

Implicit Connection Oriented Constraints

The following examples illustrate the implicit connection orientedconstraints and implicit entities used for specific informationretrieval goals. The first two examples used the terminology of entitiesand relationships.

In one example, the user wants the role (intent) played by a specifiedactor/personality (e.g., Michelle Pfeiffer) in an unspecified movie thatis about a specified role (e.g., the character Tony Montana.) In thiscase, the user's constraint includes an unspecified or implicit entity.The implicit entity is the movie ‘Scarface’. FIG. 7 illustrates a graph700 of the entities and relationships analyzed by the techniquesdisclosed herein to arrive at the desired result. The graph 700 is anillustrative visual representation of a structured informationrepository. Specifically, the implicit movie entity ‘Scarface’ 705 isarrived at via a ‘Acted In’ relationship 710 between the movie entity‘Scarface’ 705 and the actor entity ‘Michelle Pfeiffer’ 715 and a‘Character In’ relationship 720 between the character entity ‘TonyMontana’ 725 and the movie entity ‘Scarface’ 705. The role entity‘Elvira Hancock’ 730 played by ‘Michelle Pfeiffer’ is then discovered bythe ‘Acted by’ relationship 735 to ‘Michelle Pfeiffer’ and the‘Character In’ relationship 740 to the movie entity ‘Scarface’ 705.

In a further example, suppose that the user wants the movie (intent)starring the specified actor entity Scarlett Johansson and theunspecified actor entity who played the specified role of Obi-Wan Kenobiin a specified movie entity Star Wars. In this case, the implicit entityis the actor entity ‘Ewan McGregor’ and the resulting entity is themovie ‘The Island’ starring ‘Scarlett Johansson’ and ‘Ewan McGregor’.FIG. 8 illustrates a graph 800 of the entities and relationshipsanalyzed by the techniques disclosed herein to arrive at the desiredresult. Specifically, the implicit actor entity Ewan McGregor 805 isarrived at via an Acted In relationship 810 with at least one movieentity Star Wars 815 and via a Character relationship 820 to a characterentity Obi-Wan Kenobi 825, which in turn is related via a Characterrelationship 830 to the movie entity Star Wars 815. Meanwhile, theresult entity The Island 835 is arrived at via an Acted In relationship840 between the actor/personality entity Scarlett Johansson 845 and themovie entity The Island 835 and an Acted In relationship 850 between theimplicit actor entity Ewan McGregor 805 and the movie entity The Island.

FIG. 9 illustrates a graph 900 of the entities and relationshipsanalyzed by the techniques disclosed herein to arrive at a desiredresult. This example uses the terminology of nodes and edges. The userknows that there is a band that covered a Led Zeppelin song for a newmovie starring Daniel Craig. The user recalls neither the name of thecovered song nor the name of the movie, but he wants to explore theother music (i.e., songs) of the band that did that Led Zeppelin cover.Thus, by specifying the known entities of Led Zeppelin (as the songcomposer) and Daniel Craig (as an actor in the movie), the interposingimplied nodes are discovered to find the user's desired result. Thus,embodiments of the inventive techniques herein compose the queryconstraint as follows: Return the nodes of type Song (intent) connectedby a ‘Composer’ edge 905 to an implicit node of type Band 910 (TrentReznor) such that this Band node has a ‘Cover Performer’ edge 915 withan implicit node of type Song 920 (Immigrant Song) which in turn has a‘Composer’ edge 925 with an explicit node of type Band named ‘LedZeppelin’ 930 and also a ‘Track in Album’ edge 935 with an implicit nodeof type Album 940 (Girl with the Dragon Tattoo Original Sound Track)which has an ‘Original Sound Track (OST)’ edge 945 with an implicit nodeof type Movie 950 (Girl with the Dragon Tattoo Original Sound Track)that has an ‘Acted In’ edge 955 with the explicit node of typePersonality named ‘Daniel Craig’. 960.

As mentioned above, known techniques and systems for informationretrieval suffer from a variety of problems. Described herein areembodiments of an inventive conversational interaction interface. Theseembodiments enable a user to interact with an information retrievalsystem by posing a query and/or instruction by speaking to it and,optionally, selecting options by physical interaction (e.g., touchinginterface, keypad, keyboard, and/or mouse). Response to a user query maybe performed by machine generated spoken text to speech and may besupplemented by information displayed on a user screen. Embodiments ofthe conversation interaction interface, in general, allow a user to posehis next information retrieval query or instruction in reaction to theinformation retrieval system's response to a previous query, so that aninformation retrieval session is a sequence of operations, each of whichhas the user first posing a query or instruction and the system thepresenting a response to the user.

Embodiments of the present invention are a more powerful and expressiveparadigm than graphical user interfaces for the query-constraintsdiscussed herein. In many situations, especially when it comes toflexibly selecting from among a large number of possible attributes orthe presence of explicit and implicit connected nodes, the graphicaluser interface approach does not work well or does not work at all. Insuch cases, embodiments of the conversational interaction interface ofthe present invention are a much more natural fit. Further, embodimentsof the present invention are more scalable in terms of the number ofdistinct attributes a user may specify as well as the number of explicitconnected node constraints and the number of implicit node constraintsrelative to graphical user interfaces.

Conversational System Architecture

FIG. 10 represents the overall system architecture 1000 of an embodimentof the present invention. User 1001 speaks his or her question that isfed to a speech to text engine 1002. While the input could be speech,the embodiment does not preclude the input to be direct text input. Thetext form of the user input is fed to session dialog content module1003. This module maintains state across a conversation session, a keyuse of which is to help in understanding user intent during aconversation, as described below.

The session dialog content module 1003, in conjunction with a LanguageAnalyzer 1006, a Domain Specific Named Entity Recognizer 1007, a DomainSpecific Context and Intent Analyzer 1008, a Personalization BasedIntent Analyzer 1009, a Domain Specific Graph Engine 1010, and anApplication Specific Attribute Search Engine 1011 (all described in moredetail below) process the user input so as to return criteria to a QueryExecution Engine 1004. The Query Execution Engine 1004 uses the criteriato perform a search of any available source of information and contentto return a result set.

A Response Transcoding Engine 1005, dispatches the result set to theuser for consumption, e.g., in the device through which user isinteracting. If the device is a tablet device with no displayconstraints, embodiments of the present invention may leverage off thedisplay to show a graphical rendition of connection similar in spirit toFIGS. 7, 6, 9, and 8, with which the user can interact with to expressintent. In a display-constrained device such as a smartphone, theResponse Transcoding Engine 105 may respond with text and/or speech(using a standard text to speech engine).

While FIG. 10 is a conversation architecture showing the modules for aspecific domain, the present embodiment is a conversation interface thatcan take user input and engage in a dialog where user's intent can spandomains. In an embodiment of the invention, this is accomplished byhaving multiple instances of the domain specific architecture shown inFIG. 10, and scoring the intent weights across domains to determine userintent. This scoring mechanism is also used to implicitly determineconversation topic switching (for example, during a entertainmentinformation retrieval session, a user could just say “I am hungry”).

FIG. 11 illustrates a simplified flowchart of the operation ofembodiments of the invention. First, the user's speech input isconverted to text by a speech recognition engine 1101. The input is thenbroken down into intent, entities, and attributes 1102. This process isassisted by information from the prior conversation state 1103. Thebreakdown into intents, entities, and attributes, enables the system togenerate a response to the user 1104. Also, the conversation state 1103is updated to reflect the modifications of the current user input andany relevant returned response information.

FIG. 12 illustrates the control flow in more detail. First, the user'sspeech is input to the process as text 1201. Upon receiving the userinput as text, query execution coordination occurs 1202. The queryexecution coordination 1202 oversees the breakdown of the user input tounderstand user's input. The query execution coordination 1202 makes useof language analysis 1203 that parses the user input and generates aparse tree. The query execution coordination 1202 also makes use of themaintenance and updating of the dialog state 1208. The parse tree andany relevant dialog state values are passed to modules that performintent analysis 1204, entity analysis 1205, and attribute analysis 1206.These analysis processes work concurrently, because sequentialprocessing of these three analysis steps may not be possible. Forinstance, in some cases of user input, the recognition of entities mayrequire the recognition of intents and vice versa. These mutualdependencies can only be resolved by multiple passes on the input by therelevant modules, until the input is completely analyzed. Once thebreakdown and analysis is complete, a response to the user is generated1207. The dialog state is also updated 1208 to reflect the modificationsof the current input and return of relevant results. In other words,certain linguistic elements (e.g., spoken/recognized words and/orphrases) are associated with the present conversation session.

Referring again to FIG. 10, in one illustrative embodiment, the SessionDialog Content Module 1003, in conjunction with a Language Analyzer1006, and the other recognizer module, analyzer modules, and/or enginesdescribed in more detail below, perform the analysis steps mentioned inconnection with FIG. 12 and break down the sentence into its constituentparts. The Language Analyzer 1006 creates a parse tree from the textgenerated from the user input, and the other recognizer module, analyzermodules, and/or engines operate on the parse tree to determine theconstituent parts. Those parts can be broadly categorized as (1)intents—the actual intent of the user (such as “find a movie”, “play asong”, “tune to a channel”, “respond to an email”, etc.), (2)entities—noun or pronoun phrases describing or associated with theintent, and (3) attributes—qualifiers to entities such as the “latest”movie, “less” violence, etc. Other constituent part categories arewithin the scope of the invention.

In the context of the goal of providing an intelligent and meaningfulconversation, the intent is among the most important of all threecategories. Any good search engine can perform an information retrievaltask fairly well just by extracting the entities from a sentence—withoutunderstanding the grammar or the intent. For instance, the followinguser question, “Can my daughter watch pulp fiction with me”—most searchengines would show a link for pulp fiction, which may suffice to findthe rating that is most likely available from traversing that link. Butin a conversational interface, the expectation is clearly higher—thesystem must ideally understand the (movie, rating) intent correspondingto the expected response of the rating of the movie and the age group itis appropriate for. A conversational interface response degenerating tothat of a search engine is tantamount to a failure of the system from auser perspective. Intent determination and, even more importantly,responding to user's question that appears closer to a human's responsewhen the intent is not known or clearly discernible is an importantaspect for a conversational interface that strives to be closer to humaninteraction than to a search engine.

In this example, although the user never used the word “rating”, thesystem infers that user is looking for rating, from the words “can . . .watch” based on a set of rules and/or a naïve Bayes classifier,described in more details below. Meanwhile, “my daughter” could berecognized as an attribute. In order for the daughter to watch aprogram, several criteria must be met: the show timing, the showavailability, and “watchability” or rating. This condition may betriggered by other attributes too such as “son”, “girl”, “boy” etc.These could be rules-based domain specific intents or naïve Bayesclassifier scoring based on domain specific training sets to look forratings and show timings in this case. There could also be weightagefactor for the satisfiability of these conditions that is driven by theentity that is being watched.

Intent Analyzer 1008 is a domain specific module that analyzes andclassifies intent for a domain and works in conjunction with othermodules—domain specific entity recognizer 1007, personalization basedintent analyzer 1009 that classifies intent based on user's personalpreferences, and the domain specific graph engine 1010. The attributespecific search engine 1011 assists in recognizing attributes and theirweights influence the entities they qualify.

The intent analyzer 1008, in an embodiment of the invention, is a rulesdriven intent recognizer and/or a naïve Bayes classifier with supervisedtraining. The rules and/or training set capture how various words andword sets relate to user intent. It takes as input a parse tree, entityrecognizer output, and attribute specific search engine output(discussed above and below). In some implementations, user input may gothrough multiple entity recognition, the attribute recognition, andintent recognition steps, until the input is fully resolved. The intentrecognizer deciphers the intent of a sentence, and also deciphers thedifferences in nuances of intent. For instance, given “I would like tosee the movie Top Gun” versus “I would like to see a movie like TopGun”, the parse trees would be different. This difference assists theintent recognizer to differentiate the meaning of “like”. The rulesbased recognition, as the very name implies, recognizes sentences basedon predefined rules. Predefined rules are specific to a domain space,for example, entertainment. The naïve Bayes classifier component,however, just requires a training data set to recognize intent.

The entity recognizer 1007, using the inputs mentioned above, recognizesentities in user input. Examples of entities are “Tom cruise” in “can Iwatch a Tom Cruise movie”, or “Where Eagles Dare” in “when was WhereEagles Dare released”. In certain implementations, the entity recognizer1007 can be rules driven and/or a Bayes classifer. For example,linguistic elements such as nouns and gerunds can be designated asentities in a set of rules, or that association can arise during asupervised training process for the Bayes classifer. Entity recognitioncan, optionally, involve error correction or compensation for errors inuser input (such as errors in speech to text recognition). When an inputmatches two entities phonetically, e.g., newman, and neuman, both arepicked as likely candidates. In some embodiments, the resolution betweenthese two comes from the information gleaned from the rest of userinput, where relationships between entities may weed out one of thepossibilities. The classifying of a subset of user input as an entity isonly a weighting. There could be scenarios in which an input could bescored as both an entity and as an attribute. These ambiguities areresolved in many cases as the sentence semantics become clearer withsubsequent processing of the user input. In certain embodiments, acomponent used for resolution is the entity relationship graph. Incertain implementations, an output of the entity recognizer 1007 is aprobability score for subsets of input to be entities.

The application specific attribute search engine 1011 recognizesattributes such as “latest”, “recent”, “like” etc. Here again, therecould be conflicts with entities. For example “Tomorrow Never Dies” isan entity (a movie), and, when used in a sentence, there could be anambiguity in interpreting “tomorrow” as an attribute. The scoring oftomorrow as an attribute may be lower than the scoring of “tomorrow” aspart of “Tomorrow Never Dies” as determined by entity relationship graph(which may depend on other elements of the input, e.g., the words“movie”, “show”, “actor”, etc.). The output of the attribute searchengine 1011 is a probability score for input words similar to that ofthe output of entity recognizer 1007.

The language analyzer 1006 is a pluggable module in the architecture toenable to system to support multiple languages. While understanding thesemantics of user input is not constrained to the language analyzer 1006alone, the core modules of the architecture such as dialog contextmodule 1003 or graph engine 1010 are language independent. As mentionedearlier, the language module alone cannot do much more than analysis ofa sentence and performing tasks such a relating a pronoun to itssubject/object etc. (“The Led Zeppelin song in the OST of the recentDaniel Craig movie . . . Who performed it?”), it is ineffective inisolation to associate pronouns across exchanges. It is the interactionwith the session dialog context module 1003, that enables resolution ofpronouns across exchanges as in the following:

-   -   Q1: Who acts as obi-wan Kenobi in the new star wars?    -   A1: Ewan McGregor    -   Q2: How about his movies with Scarlet Johansson

While it may seem, at first glance, that dialog session context is asimple state maintenance of the last active entity or entities, thefollowing examples show the lurking complexity in dialog sessioncontext:

-   -   Q1: Who played the lead roles in Kramer vs. Kramer?    -   A1: Meryl Streep and Dustin Hoffman    -   Q2: How about more of his movies    -   A2: Here are some of Dustin Hoffman movies . . . [list of Dustin        Hoffman movies]    -   Q3: What about more of her movies?    -   A3: [list of movies if any]    -   Q4: What about just his early movies?    -   A4: [list of movies if any]    -   Q5: What about her recent movies?    -   A5: [list of movies if any]    -   Q6: Have they both acted again in the recent past ?    -   A6: [list of movies if any]    -   Q7: Have they both ever acted again at all?

In the example above, the entities Meryl Streep and Dustin Hoffman areindirectly referred to in six questions, sometimes together andsometimes separately. The above example also illustrates a distinctionof embodiments of the present invention from simple request responsesystems that engage in an exploratory exchange around a central theme.While the present embodiments not only resolve ambiguities in anexchange, they simultaneously facilitate an exploratory exchange withimplicit references to entities and/or intents mentioned much earlier ina conversation—something that is naturally done in rich humaninteractions. In certain embodiments, this is done through therecognition of linguistic linking elements, which are words and/orphrases that link the present user input to a previous user input and/orsystem response. Referring to the example provided above, the pronouns“his”, “hers”, and “they” are words that link the present user input toa previous user input and/or system response. Other pronouns, as well assyntactic expletives, can act as linguistic linking elements.

Whether a particular word or phrase used by the user in a later questionis a suitable or appropriate link to an entity mentioned in an earlierinput (or some other part of the earlier input) is determined byexamining the attributes of the earlier entity and the attributes of thepotential linking element. For example, “his” is a suitable link toDustin Hoffman in the example above because Dustin Hoffman is male, and“his” is a male gender pronoun. Moreover, “his” is a possessive pronoun,which is appropriate because the user is referring to movies in whichDustin Hoffman appears.

The following example illustrates user referring to an entity who is noteven explicitly specified in a prior exchange (an implicit connectionoriented constraint). In other words, the user never identifies theentity that is saved as part of the dialog state.

-   -   Q1: Which show had that physically challenged scientist alluding        to the possibility of non-carbon based life form?    -   A1: That was Stephen Hawking's discovery channel program on        aliens.    -   Q2: Was he in another show that David Attenborough produced?

In another example of dialog state maintenance not being restricted tojust entities and their attributes, the system can maintain state ofintents across conversation exchanges, as show as follows:

-   -   Q1: “Can my daughter watch pulp fiction with me?”    -   A1: Pulp fiction by Quentin Tarantino is rated R for graphic        violence and nudity.    -   Q2: What about his other movies?    -   A2: They are all rated R—Reservoir Dogs, Jackie Brown, Kill        Bill, Death Proof.

In this example in addition to maintain state of the entity “QuentinTarantino” which enables the system to understand the pronoun referenceto him (in the form of “his”) in Q2, the system also keeps track of userintent across the exchanges—the user intent being the “rating”. Again,the system's decision to maintain both “Quentin Taratino” and the“rating” intent stems from the rules and/or Bayes classifier trainingsets. Thus, the techniques disclosed herein enable the preservation anduse of multiple intents. In such an implementation, the set intentswould be passed as a collection of intents with weights. Depending onthe output of the rules and/or Bayes classifier, the system may elect tosave all intents during a session (and/or entities, attributes, etc.),but may only use the one intent that is scored highest for a particularinput. Thus, it is possible that an intent that accrued relativelyearlier in the dialog exchange applies much later in the conversation.Maintaining the state in this way facilitates a succinct and directedresponse as in A2, almost matching a human interaction.

The directed responses illustrated above are possible with the intentanalyzer 1008 and entity recognizer 1009 working in close concert withthe personalization based intent analyzer 1009. These modules are allassisted by an application specific attribute search engine 1011 thatassists in determining relevant attributes (e.g., latest, less ofviolence, more of action) and assigns weights to them. So a user inputexchange would come from the speech to text engine 1002, would beprocessed by the modules, analyzers, and engines working in concert(with the query execution engine 1004 playing a coordinating role), andwould yield one or more candidate interpretations of the user input. Forinstance the question, “Do you have the Kay Kay Menon movie about theBombay bomb blasts?”, the system may have two alternative candidaterepresentations wherein one has “Bombay” as an entity (there is a moviecalled Bombay) with “bomb blast” being another and the other has “Bombaybomb blast” as a single entity in another. The system then attempts toresolve between these candidate representations by engaging in a dialogwith the user, on the basis of the presence of the other recognizedentity Kay Kay Menon who is an actor. In such a case, the question(s) toformulate depend on the ambiguity that arises. In this example, theactor entity is known, it is the associated movie entities that areambiguous. Thus, the system would ask questions concerning the movieentities. The system has a set of forms that are used as a model to formquestions to resolve the ambiguities.

In some instances, resolution of ambiguity can be done, without engagingin a dialog, by knowing user's preferences. For instance, the user mayask “Is there a sox game tonight?” While this question has an ambiguousportion—the ambiguity of the team being the Boston Red Sox or theChicago White Sox—if the system is aware that user's preference is RedSox, then the response can be directed to displaying a Red Sox gameschedule if there is one that night. In instances where there aremultiple matches across domains, the domain match resulting in thehigher overall confidence score will win. Personalization of results canalso be done, when applicable, based on the nature of the query. Forinstance, if the user states “show me movies of Tom Cruise tonight”,this query should not apply personalization but just return latestmovies of Tom Cruise. However if user states “show me sports tonight”,system should apply personalization and display sports and games thatare known to be of interest to the user based on his explicitpreferences or implicit actions captured from various sources of useractivity information.

A user preference signature can be provided by the system using knowntechniques for discovering and storing such user preference information.For example, the methods and systems set forth in U.S. Pat. No.7,774,294, entitled Methods and Systems for Selecting and PresentingContent Based on Learned Periodicity of User Content Selections, issuedAug. 10, 2010, U.S. Pat. No. 7,835,998, entitled Methods and Systems forSelecting and Presenting Content on a First System Based on UserPreferences Learned on a Second System, issued Nov. 16, 2010, U.S. Pat.No. 7,461,061, entitled User Interface Methods and Systems for Selectingand Presenting Content Based on User Navigation and Selection ActionsAssociated with the Content, issued Dec. 2, 2008, and U.S. Pat. No.8,112,454, entitled Methods and Systems for Ordering Content ItemsAccording to Learned User Preferences, issued Feb. 7, 2012, each ofwhich is incorporated by reference herein, can be used with thetechniques disclosed herein. However, the use of user's preferencesignatures and/or information is not limited to the techniques set forthin the incorporated applications.

The relationship or connection engine 1010 is one of the modules thatplays a role in comprehending user input to offer a directed response.The relationship engine could be implemented in many ways, a graph datastructure being one instance so that we may call the relationship engineby the name graph engine. The graph engine evaluates the user input inthe backdrop of known weighted connections between entities.

One embodiment showing the importance of the graph engine is illustratedby the following example in which user intent is clearly known. If theuser simply queries ‘what is the role played by Michelle Pfeiffer in theTony Montana movie’, the system knows the user intent (the word role andits usage in the sentence may be used to deduce that the user wants toknow the character that Michelle Pfeiffer has played somewhere) and hasto grapple with the fact that the named entity Tony Montana could be theactor named Tony Montana or the name of the leading character of themovie Scarface. The graph engine in this instance is trivially able todisambiguate since a quick analysis of the path between the two TonyMontana entities respectively and the entity of Michelle Pfeifferquickly reveals that the actor Tony Montana never collaborated withMichelle Pfeiffer, whereas the movie Scarface (about the character TonyMontana) starred Michelle Pfeiffer. Thus, the system will conclude thatit can safely ignore the actor Tony Montana and that the user wants toknow the name of the character played by Michelle Pfeiffer in the movieScarface.

In another embodiment, the graph engine 1010 assists when the system isunable to determine the user intent despite the fact that the entityrecognizer 1007 has computed the entities specified by the user. This isillustrated by the following examples in which the user intent cannot beinferred or when the confidence score of the user intent is below athreshold. In such a scenario, two illustrative strategies could betaken by a conversation system to get the user's specific intent. Insome embodiments, the system determined the most important keywords fromthe user utterance and treats each result candidate as a document,calculates a relevance score of each document based on the eachkeyword's relevance, and presents the top few documents to the user forhim to peruse. This approach is similar to the web search engines. Inother embodiments, the system admits to the user that it cannot processthe user request or that the information it gathered is insufficient,thereby prompting the user to provide more information or a subsequentquery.

However, neither approach is entirely satisfactory when one considersthe response from the user's perspective. The first strategy, which doesblind keyword matches, can often look completely mechanical. The secondapproach attempts to be human-like when it requests the user in ahuman-like manner to furnish more information to make up for the factthat it could not compute the specific user-intent. However, in thecases that the user clearly specifies one or more other entities relatedto the desired user intent, the system looks incapable if the systemappear to not attempt an answer using the clearly specified entities inthe user utterance.

In certain implementations, a third strategy is employed so long asentity recognition has succeeded (even when the specific user intentcalculation has failed). Note that entity recognition computation issuccessful in a large number of cases, especially when the user names orgives very good clues as to the entities in his utterance, which isusually the case.

The strategy is as follows:

-   -   1. Consider the entity relationship graph corresponding to the        information repository in question. Entities are nodes and        relationships are edges in this graph. This mapping involving        entities/nodes and relationships/edges can involve one-to-one,        one-to-many, and many-to-many mapping based on information and        metadata associated with the entities being mapped.    -   2. Entities/nodes have types from a finite and well-defined set        of types.    -   3. Since entity recognition is successful (e.g., from an earlier        interaction), we consider the following cases:        -   a. Number of presently recognized entities is 0: In this            case, the system gives one from a fixed set of responses            based on response templates using the information from the            user that is recognized. The template selections is based on            rules and/or Bayes classifier determinations.        -   b. Number of recognized entities is 1: Suppose that the            entity identifier is A and the type of the entity is B and            we know the finite set S of all the distinct            edge/relationship types A can be involved in. In this case,            a system employing the techniques set forth herein (“the IR            system”) speaks and/or displays a human-consumption            multi-modal response template T(A,B,S) that follows from an            applicable template response based on A, B and S. The            template response is selected from a set of manually            constructed template responses based on a priori knowledge            of all possible node types and edge types, which form finite            well-defined sets. The response and IR system is designed to            allow the user to select, using a touch interface or even            vocally, more information and entities related to A.        -   c. Number of recognized edge types is 2: In this case, let            the two entity nodes respectively have identifiers A, A′,            types B, B′ and have edge-type sets S, S′.        -    If the edge distance between the two entity nodes is            greater than some previously decided threshold k, then the            IR system appropriately employs and delivers (via speech            and/or display) the corresponding two independent            human-consumption multi-modal response templates T(A, B, S)            and T(A′, B′, S′).        -    If the edge distance is no more than k, then the IR system            selects a shortest edge length path between A and A′. If            there are clues available in user utterance, the IR system            may prefer some shortest paths to others. Let there be k′            nodes in the selected shortest path denoted A=A₁, A₂, A₃, .            . . A_(k′)=A′ where k′<k+1 and for each i, where i goes from            1 to k′, the ith entity node of the path is represented by            the 3-tuple A_(i), B_(i) where A_(i) is the entity            identifier, B_(i) is the entity type and E_(i) is a list of            one or two elements corresponding to the one or two edges            connected to A_(i) that are present in the selected shortest            path. In this case the IR system then delivers to the user            an appropriate response based on an intelligent composition            of the sequence of human-consumption multi-modal response            templates T(A_(i), B_(i), E_(i)) where i goes from 1 to k′.        -   d. Number of recognized edge types is R>=3: In this case the            IR system simply calculates K maximal components C₁, C₂, . .            . C_(k) where each component C_(i) is such each entity node            A in C_(i) is at a distance of no more than k edges away            from at least one other node A′ of C_(i). For each C_(i),            the IR System selects an appropriate representative sequence            of human-consumption multi-modal response template            sequences, similar to c. above and composes a response based            on the response template sequences for each component.

This method to generate a response is suggested to be more human in thatit has the ability to demonstrate to the user that, with the help of theentities recognized, it presented to the user a response which made itpotentially easier compared to the two earlier strategies vis-à-vis hisgoal of retrieving information. FIGS. 7, 6, 9, and 8, illustrateexamples implementations of the disclosed techniques.

The techniques set forth above are also used, in certainimplementations, to reset all or part of the conversation state values.For example, assume a system has retained certain entities and/orattributes from user input and system responses. When the user providessubsequent input, the techniques disclosed herein enable the new inputto be evaluated against the retained values. Speaking in terms of agraph model, if linguistic elements of the subsequent input are found inan entity/relationship graph to be too far removed from the retainedinformation (also in the graph), it can be inferred that the user'ssubsequent intent has changed from the previous one. In such a case, theearlier retained information can be reset and/or disregarded whenperforming the subsequent search.

Further still, embodiments of the invention can recognize that a userhas provided subsequent input that lacks entities, attributes, orrelationship information, but the input is an appropriate response to anearlier system response. For example, a system implementing thetechniques set forth herein may present a set of movies as a response toa first user input. The user may then respond that she is not interestedin any of the movies presented. In such a case, the system would retainthe various conversation state values and make a further attempt tosatisfy the user's previous request (by, e.g., requesting additionalinformation about the type of movie desired or requesting additionalinformation to better focus the search, such as actor names, genre,etc.).

In the foregoing description, certain steps or processes can beperformed on particular servers or as part of a particular engine. Thesedescriptions are merely illustrative, as the specific steps can beperformed on various hardware devices, including, but not limited to,server systems and/or mobile devices. Similarly, the division of wherethe particular steps are performed can vary, it being understood that nodivision or a different division is within the scope of the invention.Moreover, the use of “analyzer”, “module”, “engine”, and/or other termsused to describe computer system processing is intended to beinterchangeable and to represent logic or circuitry in which thefunctionality can be executed.

The techniques and systems disclosed herein may be implemented as acomputer program product for use with a computer system or computerizedelectronic device. Such implementations may include a series of computerinstructions, or logic, fixed either on a tangible medium, such as acomputer readable medium (e.g., a diskette, CD-ROM, ROM, flash memory orother memory or fixed disk) or transmittable to a computer system or adevice, via a modem or other interface device, such as a communicationsadapter connected to a network over a medium.

The medium may be either a tangible medium (e.g., optical or analogcommunications lines) or a medium implemented with wireless techniques(e.g., Wi-Fi, cellular, microwave, infrared or other transmissiontechniques). The series of computer instructions embodies at least partof the functionality described herein with respect to the system. Thoseskilled in the art should appreciate that such computer instructions canbe written in a number of programming languages for use with manycomputer architectures or operating systems.

Furthermore, such instructions may be stored in any tangible memorydevice, such as semiconductor, magnetic, optical or other memorydevices, and may be transmitted using any communications technology,such as optical, infrared, microwave, or other transmissiontechnologies.

It is expected that such a computer program product may be distributedas a removable medium with accompanying printed or electronicdocumentation (e.g., shrink wrapped software), preloaded with a computersystem (e.g., on system ROM or fixed disk), or distributed from a serveror electronic bulletin board over the network (e.g., the Internet orWorld Wide Web). Of course, some embodiments of the invention may beimplemented as a combination of both software (e.g., a computer programproduct) and hardware. Still other embodiments of the invention areimplemented as entirely hardware, or entirely software (e.g., a computerprogram product).

What is claimed is:
 1. A computer-implemented method of inferring achange of a conversation session during continuous user interaction withan interactive content providing system having one or more processors,the method comprising: providing access to a set of content items, eachof the content items having associated metadata stored in anelectronically readable medium that describes the corresponding contentitem; receiving at one or more of the processors a first input from auser, the first input including linguistic elements intended by the userto identify at least one desired content item; associating by one ormore of the processors at least one linguistic element of the firstinput with a first conversation session; providing by one or more of theprocessors a first response based on the first input and based on themetadata associated with the content items; receiving at one or more ofthe processors a second input from the user; inferring by one or more ofthe processors whether or not the second input from the user is relatedto the at least one linguistic element associated with the firstconversation session; upon a condition in which the second input isinferred to relate to the at least one linguistic element associatedwith the first conversation session, providing by one or more of theprocessors a second response based on the metadata associated with thecontent items, the second input, and the at least one linguistic elementof the first input associated with the first conversation session; andupon a condition in which the second input is inferred to not relate tothe at least one linguistic element associated with the firstconversation session, providing by one or more of the processors asecond response based on the metadata associated with the content itemsand the second input.
 2. The method of claim 1, wherein the inferring bythe one or more processors whether or not the second input from the useris related to the at least one linguistic element associated with thefirst conversation session is based on a determination of whether or notthe second input contains a linguistic linking element.
 3. The method ofclaim 2, wherein the linguistic linking element is at least one of apronoun and a syntactic expletive.
 4. The method of claim 2, wherein thelinguistic linking element is at least one of a noun and a named entity.5. The method of claim 2, wherein the inferring by the one or moreprocessors concludes the second input from the user is related upon acondition in which the second input contains a linguistic linkingelement.
 6. The method of claim 5, wherein the linguistic linkingelement is at least one of a pronoun and a syntactic expletive.
 7. Themethod of claim 5, wherein the linguistic linking element is at leastone of a noun and a named entity.
 8. The method of claim 2, wherein theinferring by the one or more processors concludes the second input fromthe user is not related upon a condition in which the second input doesnot contain a linguistic linking element.
 9. The method of claim 8,wherein the linguistic linking element is at least one of a pronoun anda syntactic expletive.
 10. The method of claim 8, wherein the linguisticlinking element is at least one of a noun and a named entity.
 11. Themethod of claim 2, further comprising, upon a condition in which thesecond input does not contain a linguistic linking element, thefollowing: determining by the one or more processors a measure ofrelatedness between (i) linguistic elements of the second input and (ii)the at least one linguistic element associated with the firstconversation session based on the metadata associated with the contentitems; upon a condition in which the measure of relatedness is equal toor greater than a threshold value, the inferring by the one or moreprocessors concludes the second input is related to the at least onelinguistic element associated with the first conversation session; andupon a condition in which the measure of relatedness is less than thethreshold value, the inferring by the one or more processors concludesthe second input is not related to the at least one linguistic elementassociated with the first conversation session.
 12. The method of claim11, the metadata associated with the content items including a mappingof relationships between entities associated with the content items, andthe determining by the one or more processors the measure of relatednessincluding analyzing by the one or more processors the mapping ofrelationships.
 13. The method of claim 1, wherein the inferring by theone or more processors whether or not the second input from the user isrelated to the at least one linguistic element associated with the firstconversation session includes: identifying by the one or more processorsa linguistic element associated with the first conversation session thatidentifies at least one entity; identifying by the one or moreprocessors a linguistic linking element of the second input; determiningby the one or more processors whether or not the linguistic linkingelement of the second input is a suitable link to the linguistic elementassociated with the first conversation session that identifies at leastone entity; upon a condition in which the linguistic linking element ofthe second input is a suitable link, concluding by the one or moreprocessors the second input from the user is related to the at least onelinguistic element associated with the first conversation session; andupon a condition in which the linguistic linking element of the secondinput is not a suitable link, concluding by the one or more processorsthe second input from the user is not related to the at least onelinguistic element associated with the first conversation session. 14.The method of claim 13, wherein the linguistic element associated withthe first conversation session that identifies at least one entity is atleast one of a noun and a gerund.
 15. The method of claim 14, whereinthe linguistic linking element of the second input is at least one of apronoun and a syntactic expletive.
 16. The method of claim 14, whereinthe linguistic linking element of the second input is at least one of anoun and a named entity.
 17. The method of claim 1, wherein theinferring by the one or more processors whether or not the second inputfrom the user is related to the at least one linguistic elementassociated with the first conversation session includes: determining bythe one or more processors whether or not the second input includes alinguistic element that identifies at least one entity; upon a conditionin which the second input does not include a linguistic element thatidentifies at least one entity, determining by the one or moreprocessors whether or not the second input is an appropriate response tothe first response; upon a condition in which the second input is anappropriate response to the first response, concluding by the one ormore processors the second input from the user is related to the atleast one linguistic element associated with the first conversationsession; and upon a condition in which the second input is not anappropriate response to the first response, concluding by the one ormore processors the second input from the user is not related to the atleast one linguistic element associated with the first conversationsession.
 18. The method of claim 17, wherein the linguistic elementincluded in the second input that identifies at least one entity is atleast one of a noun and a gerund.
 19. The method of claim 1, upon thecondition in which the second input is inferred to relate to the atleast one linguistic element associated with the first conversationsession, the providing by the one or more processors the second responseincluding substituting by the one or more processors the at least onelinguistic element associated with the first conversation session inplace of at least one linguistic element of the second input.
 20. Themethod of claim 1, further comprising: associating by the one or moreprocessors at least one linguistic element of the first response withthe first conversation session; and upon a condition in which the secondinput is inferred to relate to the at least one linguistic elementassociated with the first conversation session, further basing thesecond response on the at least one linguistic element of the firstresponse associated with the first conversation session.
 21. The methodof claim 20, upon the condition in which the second input is inferred torelate to the at least one linguistic element associated with the firstconversation session, the providing by the one or more processors thesecond response including substituting the at least one linguisticelement associated with the first conversation session in place of atleast one linguistic element of the second input.
 22. The method ofclaim 1, the metadata associated with the content items including amapping of relationships between entities associated with the contentitems.
 23. The method of claim 1, further comprising: determining by theone or more processors that a portion of at least one of the first inputfrom the user and the second input from the user contains an ambiguousidentifier, the ambiguous identifier intended by the user to identify,at least in part, the at least one desired content item; and inferringby the one or more processors a meaning for the ambiguous identifierbased on matching portions of the at least one of the first input fromthe user and the second input from the user to preferences of the userdescribed by a user preference signature; the providing by the one ormore processors a second response including selecting content items fromthe set of content items based on comparing the inferred meaning of theambiguous identifier with metadata associated with the content items.